So-called hybrid encoding schemes, such as H.264/AVC, are the most successful class of video compression designs. Motion-compensated prediction and subsequent encoding, or transformation, of the prediction error, or the residual error, are the basic elements of these encoding schemes. The operation of a hybrid video encoder includes optimizing many decisions to accomplish the best possible trade-off between compression rate, or rate, and distortion, or image deterioration, considering constraints with respect to encoding delay and complexity. However, due to the use of motion-compensated prediction, or forecast, all these decisions typically depend on each other across many images, or frames, of an encoded sequence.
This means that the framework of the hybrid coding employed in all current video coding standards, such as MPEG-2, MPEG-4 or H.264/AVC, makes it very difficult to apply the optimization of coding decisions or coding parameters over time, that is, to consider several subsequent frames or images of a video sequence jointly or subject them jointly to an optimization. The fact that decisions in a current frame have a significant influence on the rate distortion behavior (R-D behavior) of subsequent or future frames leads to a dependently operating encoding scheme with an exponentially growing search space. Consequently, an R-D optimization is typically performed on a frame-to-frame basis. Such frame-to-frame R-D optimizations are described, for example, in A. Ortega, K. Ramchandran and M. Vetterli, “Bit Allocation for Dependent Quantization with Applications to Multiresolution and MPEG Video Coders”, IEEE Transactions on Image Processing, vol. 3, no. 5, September 1994 and G. J. Sullivan and T. Wiegand, “Rate-Distortion Optimization for Video Compression”, IEEE Signal Processing magazine, pp. 74-90, November 1998.
One approach for considering not only the current frame but the overall characteristic of a sequence is multipass encoding. In a first encoding pass, data on the statistics of the frame sequence are collected, which are then analyzed to optimize a second pass. The results from the second pass are then used for a third pass and so forth. Although multipass encoding schemes usually help to distribute the available bits more intelligently across the frame sequence, they are usually not R-D optimized.